Add WDL implementation for Salmon - #326
Conversation
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
There was a problem hiding this comment.
I didn't check out the commit, but can you comment on where these reads were sourced? We want to track the origin our our test data.
There was a problem hiding this comment.
I'd prefer if we didn't add these files. We already have FASTQ fixtures - https://github.com/stjudecloud/workflows/blob/main/test/fixtures/fastqs/README.md
The existing test fixtures should be reused (re: #280 , I don't want more LFS files hitting the history )
There was a problem hiding this comment.
I removed these customs files entirely and switched to reusing the existing shared fixtures (fastqs/test_R1.fq.gz/test_R2.fq.gz) instead and rebuilt the test transcriptome from real sequences within those files, so both build_salmon_index and quant tests now run against existing shared data rather than anything new.
There was a problem hiding this comment.
Can you rename this to use yaml as the extension to match our repository convention?
There was a problem hiding this comment.
Also this should go under tools/test/.
|
|
||
| runtime { | ||
| cpu: ncpu | ||
| memory: "16 GB" |
There was a problem hiding this comment.
Does salmon use a consistent amount of RAM or is it dependent on the input and/or transcriptome?
There was a problem hiding this comment.
Sorry, I originally hardcoded 16 GB without thinking much about how the memory usage would actually vary.
Salmon's memory usage mainly depends on the index size, which depends on the transcriptome size and whether decoys are included. The input reads are streamed, so their size doesn't have as much impact on RAM usage.
I've now updated both build_salmon_index and quant to calculate memory_gb dynamically based on the input size, similar to how disk_size_gb is already handled. There's also a modify_memory_gb option if the estimate needs to be adjusted for specific data.
| cpu: ncpu | ||
| memory: "16 GB" | ||
| disks: "~{disk_size_gb} GB" | ||
| container: "quay.io/biocontainers/salmon:1.9.0--h7e5ed60_0" |
There was a problem hiding this comment.
Is there a reason we're using such an old version of salmon?
There was a problem hiding this comment.
I'd picked 1.9.0 somewhat arbitrarily. I did try updating to 1.12.1 (the latest release still on the original C++ codebase — 2.0+ is a full Rust rewrite with a different index format, so I avoided that for now), but that specific container build (quay.io/biocontainers/salmon:1.12.1--h017bda4_0) hits a locale::facet::_S_create_c_locale crash during indexing in my test environment — a known class of bug in minimal Docker images missing locale data, unrelated to our WDL logic itself. Reverting to 1.9.0, which runs cleanly and passes both tests. Happy to revisit if you know of a working newer tag, or if this is worth filing upstream with BioContainers.
There was a problem hiding this comment.
according to the official migration doc - https://github.com/COMBINE-lab/salmon/blob/master/MIGRATION.md
I'm not seeing any reason we wouldn't want the latest version (rewrite and all). @adthrasher any reason you see not to use a >=v2 version?
| "~{if length(read_twos) == 0 then "--fldMean " + fld_mean else ""}" \ | ||
| "~{if length(read_twos) == 0 then "--fldSD " + fld_sd else ""}" \ |
There was a problem hiding this comment.
I don't think this will work as the arguments end up quoted in bash. Was this an attempt to address a sprocket lint warning?
| @@ -0,0 +1,255 @@ | |||
| version 1.1 | |||
|
|
|||
| task build_salmon_index { | |||
There was a problem hiding this comment.
The build task likely needs the decoys mode exposed as that is the recommended way to run with mapping mode.
There was a problem hiding this comment.
The information on running with decoys is scattered and not mentioned on the 2.0 doc site - https://combine-lab.github.io/salmon/
I was able to find this on the old docs site - https://salmon.readthedocs.io/en/latest/salmon.html#preparing-transcriptome-indices-mapping-based-mode
and this SC guide that seems to be where @PriyankaaXD pulled the current Bash from - https://combine-lab.github.io/alevin-tutorial/2019/selective-alignment/
I think decoy-aware indexing will be a must for any production workflow, but it seems like a barrel of worms that we can address in a follow up PR. I think we'll probably need a separate WDL task for doing the decay-aware ref building, and for this PR we can merge without touching decoys at all.
Unless @adthrasher is there a straightforward solution I'm missing?
| -l "~{lib_type}" \ | ||
| -1 ~{sep(" ", squote(read_one_fastqs_gz))} \ | ||
| ~{if length(read_twos) > 0 then "-2 " + sep(" ", squote(read_twos)) else ""} \ | ||
| --validateMappings \ |
There was a problem hiding this comment.
This is the default right? This probably needs to be a Boolean input with a true default.
| - Name: salmon_index.tar.gz | ||
|
|
||
| quant: | ||
| - name: quantifies_paired_end_reads |
There was a problem hiding this comment.
Since SE mode is implemented, it should get a test.
There was a problem hiding this comment.
I added a single-end test — and it actually caught a real bug: quant was always using -1/-2 regardless of read type, which Salmon rejects for genuine single-end input (it requires -r instead). Fixed the command logic to switch based on whether read_two_fastqs_gz is provided.
There was a problem hiding this comment.
The index here also needs documentation on how it was generated.
|
|
||
| input { | ||
| File salmon_index_tar_gz | ||
| Array[File] read_one_fastqs_gz |
There was a problem hiding this comment.
| Array[File] read_one_fastqs_gz | |
| Array[File]+ read_one_fastqs_gz |
This needs to be non-empty.
There was a problem hiding this comment.
@adthrasher I think we stopped using non-empty arrays as the resulting WDL is unwieldy ? Or we had a commit adding them and then removing them? I can't remember where we landed on it, but I'm fine without this. If the user doesn't supply any FASTQs, salmon will blow up with an informative error, so 🤷♀️
There was a problem hiding this comment.
I can't remember the details now. It's something we should probably revisit, though. I'd much rather the WDL fail upfront at analysis because of an empty array than the underlying tool erroring. If WDL doesn't do non-empty arrays well, then we should push for updates to the spec and to the engine(s).
There was a problem hiding this comment.
I'll investigate 🫡
in the mean time, let's follow through with Andrew's original recommendation here and make it non-empty
|
This also needs a |
…d FASTQ data, add SE test
Adds a WDL implementation for Salmon (mapping-mode only), per the "tool wishlist" issue #228.
Tasks added in
tools/salmon.wdl:build_salmon_index— wrapssalmon indexquant— wrapssalmon quantAll "important options" from Salmon's docs are exposed as inputs, with defaults verified against
salmon quant --help-readsoutput on Salmon 1.9.0. Parameter documentation is copied from Salmon's official docs, per guidance in the issue. Scoped to mapping-mode only (FASTQ input) — no BAM/alignment-mode support, as requested.Tests added in
tools/salmon.ymlusing the new Sprocket test framework, covering both tasks with real output assertions. Verified locally:sprocket lintpasses cleanly,sprocket dev testpasses both tests.This is my first contribution to this project — happy to make any adjustments you'd like!
Before submitting this PR, please make sure:
scripts/ordocker/directories, please ensure any image versions have been incremented accordingly!